The Next Generation of the Bar Exam: Quarterly Update

This article originally appeared in The Bar Examiner print edition, Spring 2022 (Vol. 90, No. 1), pp. 59–60.By Kellie Early

Coming into Focus

NCBE’s vision for the next generation of the bar exam is coming into clearer focus as we complete work to define the scope of the content to be assessed, plan for initial pilot testing of prototypes, and decide on the method of computer delivery.  

Content Scope Outlines

The Content Scope Outlines for the new exam were published and opened for public comment from March 25 to April 18, 2022. More than 700 stakeholders—including bar examiners and admission staff; judges and justices; law school deans, faculty, and administrators; law students; and practicing attorneys—reviewed the outlines, and individuals submitted 394 comments. We greatly appreciate the time spent by so many of you to review and provide feedback on the outlines.

For now, the draft Content Scope Outlines remain on our website. Once we complete our internal review of the comments, the outlines will be revised if necessary and stakeholders notified when the final version is published. The outlines will be used to develop the Test Content Specifications, which will be published in late 2024 after pilot and field testing data has been analyzed. Additional information helpful to candidates preparing for the exam will be incorporated into the Test Content Specifications, such as the sources of law for the topics tested, the weighting or emphasis of the subjects/topics and skills, annotations about material covered within subjects/topics, and sample test questions illustrating how the knowledge and skills may be tested.

Item Prototyping and Pilot Testing

A few item set prototypes are ready for pilot testing! We plan to do some limited pilot testing of item sets this summer before expanding these efforts in the fall. During pilot testing, some students will be provided with relevant legal resources (e.g., redacted rules), and some will be instructed to answer without access to resources so that performance variances in and between the two groups can be analyzed.

Integrated item sets will require candidates to perform one or more legal skills in the context of a realistic scenario raising one or more substantive or procedural legal issues. An item set is a collection of test questions based on a single scenario or stimulus such that the questions pertaining to that scenario are developed and presented as a unit. Item sets can be assembled so that all items within a set are either of the same format (e.g., selected response) or of different formats (e.g., selected response and short, medium, or long constructed response).

We are not yet able to show the full prototypes, as we are keeping them secure to use in the upcoming pilot testing. We can, however, provide a description of one item set to give stakeholders an idea of what some questions on the new exam might look like. This item set scenario raises issues of Criminal Law and Evidence and includes a police report, a statute governing assault and battery, and an excerpt from a case interpreting the statute. Later in the scenario, a transcript of a law clerk’s interview of a potential witness is provided. The prompts or questions related to the scenario include a mix of selected response and short answer items, such as 

• What fact in the police report will be most helpful in proving the element of “knowingly”? [selected response item]
• List three avenues you would pursue to gather other evidence relevant to issue of “great bodily harm.” [short answer item]
• Will the police officer be permitted to testify in his own words about the victim’s description of the incident? [selected response item]
• Review the transcript of the witness interview and identify two professional errors the law clerk made. [short answer item]

Integrated item sets can be scored on multiple dimensions, both for demonstrating knowledge of the relevant legal doctrine and for proficiency in the applicable legal skills. Therefore, during pilot testing we will also analyze how best to grade constructed response items in integrated item sets to ensure overall reliability of the results.

Computer Delivery

Although computer testing centers would provide greater uniformity of computer equipment and the testing environment, we have heard jurisdictions’ concerns about the amount of change it would involve. Therefore, we are actively exploring the option of delivering on laptops at jurisdiction-managed sites. Laptop delivery would allow NCBE’s focus to be on the content, design, and scoring of the new exam rather than on the administrative changes that NCBE and the jurisdictions would have to make if the exam were to be delivered at testing centers. Of course, laptop delivery is not without its own challenges, and our exploration will consider how this option might be successfully implemented in collaboration with the jurisdictions.

Recent and Upcoming Events

NCBE presented to the Massachusetts Bar Admission Committee in February 2022 and at the National Association for Law Placement Annual Education Conference in April. We held a webinar for jurisdictions in early April, and NCBE’s Annual Bar Admissions Conference in late April included a plenary session and two breakout sessions on the next generation of the bar exam. NCBE has been invited to present at the Association of Academic Support Educators Conference in May 2022 and to make presentations to members of the boards of law examiners and state supreme courts in three jurisdictions this summer. We appreciate these opportunities to meet with stakeholders. To request a presentation, please send an email to ­nextgenbarexam@ncbex.org.

To stay up to date on development of the new bar exam, subscribe to updates at nextgenbarexam.ncbex.org/subscribe. 

Contact us to request a pdf file of the original article as it appeared in the print edition.